{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost Parameter Tuning for Happy Customer Bank"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 直接调用xgboost内嵌的cv寻找最佳的参数n_estimators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据说明:\n",
    "数据集共26个字段: 其中1-24列为输入特征，25-26列为输出特征。\n",
    "    1. ID - 唯一ID（不能用于预测）\n",
    "    2. Gender - 性别\n",
    "    3. City - 城市\n",
    "    4. Monthly_Income - 月收入（以卢比为单位）\n",
    "    5. DOB - 出生日期\n",
    "    6. Lead_Creation_Date - 潜在（贷款）创建日期\n",
    "    7. Loan_Amount_Applied - 贷款申请请求金额（印度卢比，INR）\n",
    "    8. Loan_Tenure_Applied - 贷款申请期限（单位为年）\n",
    "    9. Existing_EMI -现有贷款的EMI（EMI：电子货币机构许可证） \n",
    "    10. Employer_Name雇主名称\n",
    "    11. Salary_Account - 薪资帐户银行\n",
    "    12. Mobile_Verified - 是否移动验证（Y / N）\n",
    "    13. VAR5 - 连续型变量\n",
    "    14. VAR1-  类别型变量\n",
    "    15. Loan_Amount_Submitted - 提交的贷款金额（在看到资格后修改和选择）\n",
    "    16. Loan_Tenure_Submitted - 提交的贷款期限（单位为年，在看到资格后修改和选择）\n",
    "    17. Interest_Rate - 提交贷款金额的利率\n",
    "    18. Processing_Fee - 提交贷款的处理费（INR）\n",
    "    19. EMI_Loan_Submitted -提交的EMI贷款金额（INR）\n",
    "    20. Filled_Form - 后期报价后是否已填写申请表格\n",
    "    21. Device_Type - 进行申请的设备（浏览器/移动设备）\n",
    "    22. Var2 - 类别型变量\n",
    "    23. Source - 类别型变量\n",
    "    24. Var4 - 类别型变量\n",
    "\n",
    "输出：\n",
    "    25. LoggedIn - 是否login（只用于理解问题的变量，不能用于预测，测试集中没有）\n",
    "    26. Disbursed - 是否发放贷款（目标变量），1为发放贷款（目标客户）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.read_csv('train_save.csv')\n",
    "test = pd.read_csv('test_save.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((87020, 52), (37717, 51))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.shape, test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEKCAYAAADaa8itAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAFpFJREFUeJzt3Xu0JWV95vHvw02Qi4C2LtNAGmNHJTreWoImEy8oAk5oZEGCyxEGmcFkAHESHTHJsuNdEoRI4iWsQLiMw0UGtEWUQQSdOIo0F8UGWXQwQg8EWkFEGGEafvPHfg/s7j6Xorv2OWzP97PWXqfqrbdq/7ar5VlVb9VbqSokSerDZnNdgCTpV4ehIknqjaEiSeqNoSJJ6o2hIknqjaEiSeqNoSJJ6o2hIknqjaEiSerNFnNdwGx7xjOeUYsWLZrrMiRpbFxzzTU/qaoFXfrOu1BZtGgRK1asmOsyJGlsJPlx175e/pIk9cZQkST1xlCRJPXGUJEk9cZQkST1xlCRJPXGUJEk9cZQkST1xlCRJPVm3j1Rv6le/p6z5roEPQld89eHzXUJ0pOCZyqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTejDRUkvyXJCuT/CDJOUm2TrJ7kquS3JLkvCRbtb5Paeur2vZFQ8d5X2u/Ockbh9r3bW2rkhw/yt8iSZrZyEIlyULgncCSqnohsDlwKHACcHJVLQbuBY5suxwJ3FtVzwVObv1Iskfb77eAfYFPJ9k8yebAp4D9gD2At7S+kqQ5MurLX1sA2yTZAngqcCfwOuCCtv1M4MC2vLSt07bvnSSt/dyqeqiqfgSsAvZsn1VVdWtVPQyc2/pKkubIyEKlqv4PcCJwG4MwuQ+4BvhZVa1t3VYDC9vyQuD2tu/a1v/pw+3r7TNVuyRpjozy8tdODM4cdgd+DdiWwaWq9dXELlNse6Ltk9VyVJIVSVasWbNmptIlSRtplJe/Xg/8qKrWVNX/Ay4EXgXs2C6HAewC3NGWVwO7ArTtTwPuGW5fb5+p2jdQVadW1ZKqWrJgwYI+fpskaRKjDJXbgL2SPLWNjewN3AhcARzc+hwOfLEtL2/rtO1fr6pq7Ye2u8N2BxYD3wWuBha3u8m2YjCYv3yEv0eSNIORvaSrqq5KcgFwLbAWuA44FfgycG6SD7e209oupwFnJ1nF4Azl0HaclUnOZxBIa4Gjq+oRgCTHAJcyuLPs9KpaOarfI0ma2Ujf/FhVy4Bl6zXfyuDOrfX7/hI4ZIrjfAT4yCTtlwCXbHqlkqQ++ES9JKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3hookqTeGiiSpN4aKJKk3M4ZKkm2TbNaWfzPJAUm2HH1pkqRx0+VM5ZvA1kkWApcDRwBnjLIoSdJ46hIqqaoHgYOAv62qNwN7jLYsSdI46hQqSV4JvBX4cmvbYnQlSZLGVZdQeRfwPuCiqlqZ5DnAFaMtS5I0jmY846iqbwDfSLJtW78VeOeoC5MkjZ8ud3+9MsmNwE1t/cVJPj3yyiRJY6fL5a+/Ad4I/BSgqr4H/N4oi5IkjadODz9W1e3rNT0yglokSWOuy11ctyd5FVBJtmIwnnLTaMuSJI2jLmcqfwQcDSwEVgMvaeuSJK2jy91fP2HwjIokSdPqcvfXmUl2HFrfKcnpoy1LkjSOulz++jdV9bOJlaq6F3jp6EqSJI2rLqGyWZKdJlaS7IzTtEiSJtElHD4B/O8kF7T1Q4CPjK4kSdK46jJQf1aSa4DXAgEOqqobR16ZJGnsdH3z4w+BC4EvAr9IsluXnZLsmOSCJD9MclOb8mXnJJcluaX93an1TZJTkqxK8v0kLxs6zuGt/y1JDh9qf3mSG9o+pyRJ958uSepbl7u/jgXuAi4DLmYw/f3FHY//SeCrVfV84MUMHpo8Hri8qhYzeOnX8a3vfsDi9jkK+Ez7/p2BZcBvA3sCy4bGeD7T+k7st2/HuiRJI9BlTOU44HlV9dMncuAkOzCYI+w/AFTVw8DDSZYCr2ndzgSuBN4LLAXOqqoCvtPOcp7d+l5WVfe0414G7JvkSmCHqvp2az8LOBD4yhOpU5LUny6Xv24H7tuIYz8HWAP8Y5LrkvxDmz7/WVV1J0D7+8zWf2H7rgmrW9t07asnaZckzZEuZyq3Alcm+TLw0ERjVZ3U4dgvA46tqquSfJLHL3VNZrLxkNqI9g0PnBzF4DIZu+3WaThIkrQRupyp3MZgPGUrYPuhz0xWA6ur6qq2fgGDkLmrXdai/b17qP+uQ/vvAtwxQ/suk7RvoKpOraolVbVkwYIFHUqXJG2MLrcUfwAgybZV9UDXA1fVvya5PcnzqupmYG/gxvY5HPh4+/vFtsty4Jgk5zIYlL+vqu5Mcinw0aHB+X2A91XVPUnuT7IXcBVwGPC3XeuTJPVvxlBJ8krgNGA7YLckLwbeUVX/ucPxjwU+16bMvxU4gsHZ0flJjmRwFnRI63sJsD+wCniw9aWFx4eAq1u/D04M2gN/DJwBbMNggN5BekmaQ13GVCbe/LgcBm9+TNLpzY9VdT2wZJJNe0/St5hiSv2qOh3YYBLLqloBvLBLLZKk0fPNj5Kk3vjmR0lSb3zzoySpN9OeqSTZHHhbVfnmR0nSjKY9U6mqRxhMnyJJ0oy6jKl8K8nfAecBjz2nUlXXjqwqSdJY6hIqr2p/PzjUVsDr+i9HkjTOZhpT2Qz4TFWdP0v1SJLG2ExjKo8Cx8xSLZKkMdflluLLkrw7ya7trY07txdnSZK0ji5jKm9vf4efTSkG70uRJOkxXWYp3n02CpEkjb8usxQfNll7VZ3VfzmSpHHW5fLXK4aWt2Yww/C1gKEiSVpHl8tfxw6vJ3kacPbIKpIkja1OU9+v50Fgcd+FSJLGX5cxlS8xuNsLBiG0B+DDkJKkDXQZUzlxaHkt8OOqWj2ieiRJY6xLqNwG3FlVvwRIsk2SRVX1LyOtTJI0drqMqXweeHRo/ZHWJknSOrqEyhZV9fDESlveanQlSZLGVZdQWZPkgImVJEuBn4yuJEnSuOoypvJHwOfai7pg8J76SZ+ylyTNb10efvxnYK8k2wGpqvtHX5YkaRzNePkryUeT7FhVv6iq+5PslOTDs1GcJGm8dBlT2a+qfjaxUlX3AvuPriRJ0rjqEiqbJ3nKxEqSbYCnTNNfkjRPdRmo/2/A5Un+kcF0LW8HzhxpVZKksdRloP6vknwfeH1r+lBVXTrasiRJ46jLmQrAdcCWDM5UrhtdOZKkcdbl7q8/AL4LHAz8AXBVkoNHXZgkafx0OVP5c+AVVXU3QJIFwNeAC0ZZmCRp/HS5+2uziUBpftpxP0nSPNPlTOWrSS4FzmnrfwhcMrqSJEnjqsvdX+9JchDwu0CAU6vqopFXJkkaO53u/qqqC4ELR1yLJGnMOTYiSeqNoSJJ6s2UoZLk8vb3hNkrR5I0zqYbU3l2klcDByQ5l8Eg/WOq6tqRViZJGjvTXf56P3A8sAtwEvCJoc+JXb8gyeZJrktycVvfPclVSW5Jcl6SrVr7U9r6qrZ90dAx3tfab07yxqH2fVvbqiTHd//ZkqRRmDJUquqCqtoP+Kuqeu16n9c9ge84DrhpaP0E4OSqWgzcCxzZ2o8E7q2q5wInt34k2QM4FPgtYF/g0y2oNgc+BewH7AG8pfWVJM2RGQfqq+pDSQ5IcmL7/LuuB0+yC/Am4B/aeoDX8fgUL2cCB7blpTw+pf4FwN6t/1Lg3Kp6qKp+BKwC9myfVVV1a1U9DJzb+kqS5kiXCSU/xuBs48b2Oa61dfE3wH8FHm3rTwd+VlVr2/pqYGFbXgjcDtC239f6P9a+3j5TtU/2G45KsiLJijVr1nQsXZL0RHW5pfhNwBuq6vSqOp3BJag3zbRTO6O5u6quGW6epGvNsO2Jtm/YWHVqVS2pqiULFiyYpmpJ0qbo+j6VHYF72vLTOu7zOwzuHNsf2BrYgcGZy45JtmhnI7sAd7T+q4FdgdVJtmjfc89Q+4ThfaZqlyTNgS5nKh8DrktyRpIzgWuAj860U1W9r6p2qapFDAbav15VbwWuYPBuFoDDgS+25eVtnbb961VVrf3QdnfY7sBiBu93uRpY3O4m26p9x/IOv0eSNCJdJpQ8J8mVwCsYXHJ6b1X96yZ853uBc5N8mMFbJE9r7acBZydZxeAM5dD2/SuTnM9gPGctcHRVPQKQ5BjgUmBz4PSqWrkJdUmSNlHXCSXvZBPOAqrqSuDKtnwrgzu31u/zS+CQKfb/CPCRSdovwWn4JelJw7m/JEm9MVQkSb2ZNlSSbJbkB7NVjCRpvE0bKlX1KPC9JLvNUj2SpDHWZaD+2cDKJN8FHphorKoDRlaVJGksdQmVD4y8CknSr4Quz6l8I8mvA4ur6mtJnsrguRBJktbRZULJ/8Rg1uC/b00LgS+MsihJ0njqckvx0Qzm8fo5QFXdAjxzlEVJksZTl1B5qL2vBIA22eOkswFLkua3LqHyjSR/BmyT5A3A54EvjbYsSdI46hIqxwNrgBuAdzCYa+svRlmUJGk8dbn769E25f1VDC573dympJckaR0zhkqSNwGfBf6ZwdT3uyd5R1V9ZdTFSZLGS5eHHz8BvLaqVgEk+Q3gy4ChIklaR5cxlbsnAqW5Fbh7RPVIksbYlGcqSQ5qiyuTXAKcz2BM5RAGr/KVJGkd013++v2h5buAV7flNcBOI6tIkjS2pgyVqjpiNguRJI2/Lnd/7Q4cCywa7u/U95Kk9XW5++sLwGkMnqJ/dLTlSJLGWZdQ+WVVnTLySiRJY69LqHwyyTLgfwIPTTRW1bUjq0qSNJa6hMqLgLcBr+Pxy1/V1iVJekyXUHkz8Jzh6e8lSZpMlyfqvwfsOOpCJEnjr8uZyrOAHya5mnXHVLylWJK0ji6hsmzkVUiSfiV0eZ/KN2ajEEnS+OvyRP39PP5O+q2ALYEHqmqHURYmSRo/Xc5Uth9eT3IgsOfIKpIkja0ud3+to6q+gM+oSJIm0eXy10FDq5sBS3j8cpgkSY/pcvfX8HtV1gL/AiwdSTWSpLHWZUzF96pIkjqZ7nXC759mv6qqD42gHknSGJvuTOWBSdq2BY4Eng4YKpKkdUz3OuFPTCwn2R44DjgCOBf4xFT7SZLmr2nHVJLsDPwJ8FbgTOBlVXXvbBQmSRo/Uz6nkuSvgauB+4EXVdVfPpFASbJrkiuS3JRkZZLjWvvOSS5Lckv7u1NrT5JTkqxK8v0kLxs61uGt/y1JDh9qf3mSG9o+pyTJRvxvIEnqyXQPP/4p8GvAXwB3JPl5+9yf5Ocdjr0W+NOqegGwF3B0kj2A44HLq2oxcHlbB9gPWNw+RwGfgcfOlpYBv83gSf5lE0HU+hw1tN++3X62JGkUpgyVqtqsqrapqu2raoehz/Zd5v2qqjsnXjlcVfcDNwELGTzjcmbrdiZwYFteCpxVA98BdkzybOCNwGVVdU87U7oM2Ldt26Gqvl1VBZw1dCxJ0hx4wtO0bIwki4CXAlcBz6qqO2EQPMAzW7eFwO1Du61ubdO1r56kfbLvPyrJiiQr1qxZs6k/R5I0hZGHSpLtgP8BvKuqprtsNtl4SG1E+4aNVadW1ZKqWrJgwYKZSpYkbaSRhkqSLRkEyueq6sLWfFe7dEX7e3drXw3sOrT7LsAdM7TvMkm7JGmOjCxU2p1YpwE3VdVJQ5uWAxN3cB0OfHGo/bB2F9hewH3t8tilwD5JdmoD9PsAl7Zt9yfZq33XYUPHkiTNgS4TSm6s3wHeBtyQ5PrW9mfAx4HzkxwJ3AYc0rZdAuwPrAIeZPCgJVV1T5IPMbi9GeCDVXVPW/5j4AxgG+Ar7SNJmiMjC5Wq+icmH/cA2HuS/gUcPcWxTgdOn6R9BfDCTShTktSjWbn7S5I0PxgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN4YKpKk3hgqkqTeGCqSpN6Mfagk2TfJzUlWJTl+ruuRpPlsi7kuYFMk2Rz4FPAGYDVwdZLlVXXj3FYmzY3bPviiuS5BT0K7vf+GWfuucT9T2RNYVVW3VtXDwLnA0jmuSZLmrXEPlYXA7UPrq1ubJGkOjPXlLyCTtNUGnZKjgKPa6i+S3DzSquaPZwA/mesingxy4uFzXYI25L/PCcsm+0/lE/LrXTuOe6isBnYdWt8FuGP9TlV1KnDqbBU1XyRZUVVL5roOaTL++5wb437562pgcZLdk2wFHAosn+OaJGneGuszlapam+QY4FJgc+D0qlo5x2VJ0rw11qECUFWXAJfMdR3zlJcU9WTmv885kKoNxrUlSdoo4z6mIkl6EjFUNKOZpsJJ8pQk57XtVyVZNPtVaj5KcnqSu5P8YIrtSXJK+7f5/SQvm+0a5xtDRdMamgpnP2AP4C1J9liv25HAvVX1XOBk4ITZrVLz2BnAvtNs3w9Y3D5HAZ+ZhZrmNUNFM+kyFc5S4My2fAGwd5JNftpKmklVfRO4Z5ouS4GzauA7wI5Jnj071c1Phopm0mUqnMf6VNVa4D7g6bNSnTQ9p3KaZYaKZtJlKpxO0+VIc8B/m7PMUNFMukyF81ifJFsAT2P6SxLSbOk0lZP6Y6hoJl2mwlkOTMyoeDDw9fIBKD05LAcOa3eB7QXcV1V3znVRv8rG/ol6jdZUU+Ek+SCwoqqWA6cBZydZxeAM5dC5q1jzSZJzgNcAz0iyGlgGbAlQVZ9lMNvG/sAq4EHgiLmpdP7wiXpJUm+8/CVJ6o2hIknqjaEiSeqNoSJJ6o2hIknqjaEidZTkkSTXJ1mZ5HtJ/iTJZm3bkiSnTLPva5JcPHvVbvD9f5nk3XP1/Zo/fE5F6u7/VtVLAJI8E/jvDGYPWFZVK4AVo/riJFu0edWkJzXPVKSNUFV3M5hK/Zj2tPZjZyJJXt3OaK5Pcl2S7dtuOyS5KMmNST47dJbzi4njJjk4yRlt+YwkJyW5AjhhquMmeU+Sq9v7Qj4wdKw/b+/B+RrwvNn430XyTEXaSFV1awuGZ6636d3A0VX1rSTbAb9s7XsyeCfNj4GvAgcxeFXAdH4TeH1VPZLkS+sfN8k+DN4VsieDyROXJ/k94AEGMxu8lMH/z68Frtm0XyzNzDMVadNMNgvut4CTkrwT2HHostV323tpHgHOAX63w/E/3/pPddx92uc6BsHxfAYh82+Bi6rqwar6ORvO1yaNhKEibaQkzwEeAe4ebq+qjwP/EdgG+E6S509sWu8QNUn71uv1eWCG4wb4WFW9pH2eW1WnTfF90sgZKtJGSLIA+Czwd+vPyJzkN6rqhqo6gcHg/USo7Nlme94M+EPgn1r7XUle0NrfPM13TnbcS4G3t8thJFnYbiL4JvDmJNu0sZff7+u3S9NxTEXqbpsk1zOYBXctcDZw0iT93pXktQzOYm4EvgK8Evg28HHgRQz+o39R6388cDGDNxT+ANhuiu/f4LhV9VCSFwDfbm9w/gXw76vq2iTnAdczGMP5X5vyw6WunKVYktQbL39JknpjqEiSemOoSJJ6Y6hIknpjqEiSemOoSJJ6Y6hIknpjqEiSevP/AQfaa5oYJqxDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(train.Disbursed);\n",
    "pyplot.xlabel('Disbursed');\n",
    "pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "样本不均衡，使用交叉验证"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "默认参数，此时学习率为0.1，比较大，观察弱分类数目的大致范围\n",
    "（采用默认参数配置，看看模型是过拟合还是欠拟合）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train = train['Disbursed']\n",
    "train = train.drop([\"Disbursed\"], axis=1)\n",
    "X_train = np.array(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 87020 entries, 0 to 87019\n",
      "Data columns (total 51 columns):\n",
      "Existing_EMI                     87020 non-null float64\n",
      "Loan_Amount_Applied              87020 non-null float64\n",
      "Loan_Tenure_Applied              87020 non-null float64\n",
      "Monthly_Income                   87020 non-null int64\n",
      "Var4                             87020 non-null int64\n",
      "Var5                             87020 non-null int64\n",
      "Age                              87020 non-null int64\n",
      "EMI_Loan_Submitted_Missing       87020 non-null int64\n",
      "Interest_Rate_Missing            87020 non-null int64\n",
      "Loan_Amount_Submitted_Missing    87020 non-null int64\n",
      "Loan_Tenure_Submitted_Missing    87020 non-null int64\n",
      "Processing_Fee_Missing           87020 non-null int64\n",
      "Device_Type_0                    87020 non-null int64\n",
      "Device_Type_1                    87020 non-null int64\n",
      "Filled_Form_0                    87020 non-null int64\n",
      "Filled_Form_1                    87020 non-null int64\n",
      "Gender_0                         87020 non-null int64\n",
      "Gender_1                         87020 non-null int64\n",
      "Var1_0                           87020 non-null int64\n",
      "Var1_1                           87020 non-null int64\n",
      "Var1_2                           87020 non-null int64\n",
      "Var1_3                           87020 non-null int64\n",
      "Var1_4                           87020 non-null int64\n",
      "Var1_5                           87020 non-null int64\n",
      "Var1_6                           87020 non-null int64\n",
      "Var1_7                           87020 non-null int64\n",
      "Var1_8                           87020 non-null int64\n",
      "Var1_9                           87020 non-null int64\n",
      "Var1_10                          87020 non-null int64\n",
      "Var1_11                          87020 non-null int64\n",
      "Var1_12                          87020 non-null int64\n",
      "Var1_13                          87020 non-null int64\n",
      "Var1_14                          87020 non-null int64\n",
      "Var1_15                          87020 non-null int64\n",
      "Var1_16                          87020 non-null int64\n",
      "Var1_17                          87020 non-null int64\n",
      "Var1_18                          87020 non-null int64\n",
      "Var2_0                           87020 non-null int64\n",
      "Var2_1                           87020 non-null int64\n",
      "Var2_2                           87020 non-null int64\n",
      "Var2_3                           87020 non-null int64\n",
      "Var2_4                           87020 non-null int64\n",
      "Var2_5                           87020 non-null int64\n",
      "Var2_6                           87020 non-null int64\n",
      "Mobile_Verified_0                87020 non-null int64\n",
      "Mobile_Verified_1                87020 non-null int64\n",
      "Source_0                         87020 non-null int64\n",
      "Source_1                         87020 non-null int64\n",
      "Source_2                         87020 non-null int64\n",
      "Source_3                         87020 non-null int64\n",
      "Source_4                         87020 non-null int64\n",
      "dtypes: float64(3), int64(48)\n",
      "memory usage: 33.9 MB\n"
     ]
    }
   ],
   "source": [
    "train.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#直接调用xgboost内嵌的交叉验证（cv），可对连续的n_estimators参数进行快速交叉验证\n",
    "#而GridSearchCV只能对有限个参数进行交叉验证\n",
    "def modelfit(alg, X_train, y_train, cv_folds=None, early_stopping_rounds=10):\n",
    "    xgb_param = alg.get_xgb_params()\n",
    "#    xgb_param['num_class'] = 2\n",
    "    \n",
    "    #直接调用xgboost，而非sklarn的wrapper类\n",
    "    xgtrain = xgb.DMatrix(X_train, label = y_train)\n",
    "        \n",
    "    cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], folds =cv_folds,\n",
    "             metrics='logloss', early_stopping_rounds=early_stopping_rounds)\n",
    "  \n",
    "    cvresult.to_csv('1_nestimators.csv', index_label = 'n_estimators')\n",
    "    \n",
    "    #最佳参数n_estimators\n",
    "    n_estimators = cvresult.shape[0]\n",
    "    \n",
    "    # 采用交叉验证得到的最佳参数n_estimators，训练模型\n",
    "    alg.set_params(n_estimators = n_estimators)\n",
    "    alg.fit(X_train, y_train, eval_metric='logloss')\n",
    "    cvresult\n",
    "        \n",
    "    #Predict training set:\n",
    "    #train_predprob = alg.predict_proba(X_train)\n",
    "    #logloss = log_loss(y_train, train_predprob)\n",
    "\n",
    "   #Print model report:\n",
    "   # print (\"logloss of train :\" )\n",
    "   # print logloss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#params = {\"objective\": \"multi:softprob\", \"eval_metric\":\"mlogloss\", \"num_class\": 9}\n",
    "xgb1 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=1000,  #数值大没关系，cv会自动返回合适的n_estimators\n",
    "        max_depth=5,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel=0.7,\n",
    "        objective= 'binary:logistic',\n",
    "        seed=3)\n",
    "\n",
    "modelfit(xgb1, X_train, y_train, cv_folds = kfold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ice2018/anaconda2/lib/python2.7/site-packages/ipykernel_launcher.py:1: FutureWarning: from_csv is deprecated. Please use read_csv(...) instead. Note that some of the default arguments are different, so please refer to the documentation for from_csv when changing your function calls\n",
      "  \"\"\"Entry point for launching an IPython kernel.\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEXCAYAAABCjVgAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8XHW9//HXZ2ayN23aJt2SdKUFCpStrCoi4BWQTUQEQUVFf+rloiIi3J8/vXIFFe9FvRcuimxugAWRHbnIquwte1sKpS1NuqZbtjbbzOf3xzlJpyFp0pLJmWTez8fjPDJnmXM+c2Yy7znb95i7IyIiAhCLugAREckeCgUREemiUBARkS4KBRER6aJQEBGRLgoFERHpolAQSWNm/2pmN0Rdh0hUFApDjJmNMLMVZvaZtGGlZrbSzM5IGzbXzO43s81mtsXMFpnZFWY2Ohx/npklzawp7JaZ2dcyXPvRZlabyWXsip7qcfcr3f38DC1vhZkdl4l5Z8JgvV9Dbb0MdwqFIcbdm4CvAL80s4pw8FXAfHe/E8DMjgSeAJ4G9nL3MuB4oAPYP212z7r7CHcfAZwBXGVmBw7OK5FdYWaJqGuQHOHu6oZgB9wC3AYcDWwEJqaN+wfw3308/zzgH92GvQB8Jq3/FGAhsIUgZPZOG7d3OGxLOM0paeNOBBYBjcAq4GKgBNgGpICmsJvUy+u6FnggfP7zwIx+rI+9gEeATcAS4MzdqQf4N+AP4fOmAg58AagBNgNfBQ4BXgtf+zVpy5kBPBa+HxuAPwJl4bjfh8vaFi7rkn6s4xXAd8NltQKJsH9V+FqWAMf2sC4OB9YC8bRhnwBeCx8fCswHGoB1wNW9rNOjgdpexo0CfgfUAe8C3wNi4bg48J/hOlgOXBCux0Qv81oBHNfLuC8DS8P39d7OzwxgwM+B9UB9uI727e39jvr/dSh1kRegbjffOBgNrAn/8b6QNrwESAJH9/H880gLhfCLbgswK+yfBTQDHwXygEvCf878sH8p8K9h/zHhP+Ce4XPXAB9Kq/Og8HGvXzJpddwSfgEcGn4J/hG4vY/nlBB8aX8hfM5B4XrZZ1froedQ+BVQCPwT0ALcDYwDKsMvpQ+H0+8Rrq8CoAJ4CvhF2rx3+PLb2TpOm/4VoBooAvYMX+ektPp6DEzgHeCjaf13AJeGj58FPhs+HgEc3ss8en2/CALhHqA0rOMt4EvhuK8SfClXhev7b+xGKISfqw3h+1kA/DfwVDjuY8ACoIwgIPYm/GHU2/utrn+ddh8NUe6+meAXZjFwV9qo0QS7Bdd2DjCzq8LjCs1m9r20aQ8PhzcRbCX8Hng7HPdp4AF3f8Td24H/IPhiOpLgl+gI4Cfu3ubujwH3A2eHz20HZpvZSHff7O4v7eLLu8vdX3D3DoJQOKCP6U8CVrj7ze7eES7vzwS7xAainn939xZ3/1+CL/Hb3H29u68C/g4cCODuS8P11erudcDVwId3Mt+dreNO/+XuNe6+jSDsC8LXkufuK9z9nV7mfRvh+2FmpQS/nm9LWx97mFm5uze5+3O7sjLMLB7Wfpm7N7r7CoItg8+Gk5wJ/NLda8PP6U92Zf5pzgFucveX3L0VuAw4wsymhq+hlGAL0dx9sbuvSXt97+f9zmkKhSHKzM4l+IX2N+CnaaM2E+ymmNg5wN0v8eC4wl8Ifkl3es7dyzw4pjAB2Ae4Mhw3iWC3QOc8UgS/UivDcTXhsE7vhuMAPknwJfSumT1pZkfs4stbm/Z4K0EA7cwU4LAw4LaY2RaCL5QJA1TPurTH23roHwFgZuPM7HYzW2VmDcAfgPKdzHdn67hTTdr4pcA3CbZm1ofLmtTLvG8FTjezAuB04CV371zWlwi2Ut40sxfN7KSd1NiTcoItxHfThqW//5PS6+72eFd0Xz9NBLvmKsMfItcQ7GpcZ2bXm9nIcNL3+37nNIXCEGRm4wj2p34Z+D/AmWZ2FIC7NxPshz99V+bp7usIfl2fHA5aTfBl27lMI9iNsSocV21m6Z+fyeE43P1Fdz+VYBfL3cC8zsXsSk27oAZ4Mgy4zm6Eu39tkOv5cTjPOe4+EjiXYNdGp+7L29k67vE57n6ru38wfJ6z4w+C9OkWEXyhngB8hiAkOse97e5nE6yPnwJ3mllJ/18mGwh+jU9JG9b1/hPsvqlKG1e9C/NO1339lABj2f45+y93P5jgx8ws4Dvh8N7eb+kHhcLQdA1wt7s/Hm4yXwL8JvxVSNj/RTO7NAwQzKwKmNbbDM1sLMHByIXhoHnAx83sWDPLA75NcLDzGYLQaQYuMbM8MzuaIExuN7N8MzvHzEaFu0QaCHZ7QPALe6yZjRqg9dDpfmCWmX02rCfPzA4xs70HuZ5SgoPIW8yskvBLKs06YHpa/87W8XuY2Z5mdkz4PrcQbKUke5o2dCtwIXAUwTGFzvmca2YV4ZbJlnBwr/Mxs8L0jmBLdB5wRXg69BTgIoIto87X9Q0zqzSzMoKD433J67acRFj/F8zsgPA1Xwk87+4rwvf3sHC9NYfrI9nH+y39EfVBDXW71gGnEfyCKus2/FHgirT+w4AHCf7ptwBvAFcAY8Px5xH8s3SeebOeYJ/zuLR5fILggGE98CThgdtw3D7hsPpwmk+Ew/OBvxLsxmoAXgQ+mPa8mwh2AWyh97OPfpTWfzR9HJwOp9uT4IylunD+jxEci9ileuj5QHMibfpa0g7iE3wRfi9tnSwI1+crBF/ytWnTngqsDJd1cT/W8Qp2PDA9h+DYTyPBwfj7e1qHadNPJvgCf6Db8D+E73cTwY+A03p5/tHh6+/e7UFw7OoP4fquAb7P9rOPEgRbshsJzj76FsGWhfWynBU9LONH4bivEhw073y9VeHwYwnOOGpi+5leI/p6v9X13Vm4gkVEMsLMTgB+5e5T+pxYIqfdRyIyoMysyMxONLNEuBvtBwQnOcgQoC0FGRLM7EPAQz2N8+DsKckSZlZMsCtsL4LjHg8A33D3hkgLk35RKIiISBftPhIRkS5DrpGt8vJynzp1atRliIgMKQsWLNjg7hV9TTfkQmHq1KnMnz8/6jJERIYUM3u376m0+0hERNIoFEREpItCQUREuigURESki0JBRES6KBRERKRLRkPBzI43syVmttTMLu1lmjPNbJGZLTSzW3uaRkREBkfGQiG8Zd+1BDf5mA2cbWazu00zk+AWex9w930I7iqVEcsWvcDzf7oKT6X6nlhEJEdlckvhUGCpuy9z9zbgdoL25NN9GbjWg/u44u7rM1XM+pcf4rDFV9CwZWOmFiEiMuRlMhQq2fHerLXseO9ZCG6hN8vMnjaz58zs+EwVUzA2aMq9rvbtPqYUEcldmQwF62FY9yZZE8BMgjs8nQ3cEN6+b8cZmX3FzOab2fy6urrdKmbEhBkANK5dtlvPFxHJBZkMhVp2vGF3FcFtJLtPc4+7t7v7cmAJQUjswN2vd/e57j63oqLP9px6VFEVzLZ1w/Lder6ISC7IZCi8CMw0s2lmlg+cBdzbbZq7gY8AmFk5we6kjPyUHzVmHM1eCFtWZmL2IiLDQsZCwd07gAuAh4HFwDx3X2hml5vZKeFkDwMbzWwR8DjwHXfPyJFgi8VYHx9PQVNtJmYvIjIsZLTpbHd/EHiw27Dvpz124KKwy7j6gomMbF0zGIsSERmScuqK5taSSVQkM3bWq4jIkJdToeBlUxhJM/WbN0RdiohIVsqpUMgPr1XYULs04kpERLJTToVCaXitQsMahYKISE9yKhTKu65VWBFtISIiWSqnQqFs7Hi2eoGuVRAR6UVOhULntQr5TauiLkVEJCvlVCgA1BdM0LUKIiK9yLlQaCmpoiK5NuoyRESyUs6Fgo+qZhTNuq+CiEgPci4U8sZOA3StgohIT3IuFEonTAegYc07EVciIpJ9ci4Uyqv2AKBF1yqIiLxHzoXC6PKJbPUCXNcqiIi8R86FgsVi1MXH6b4KIiI9yLlQAKjPn8DIlu53BhURkZwMhW0llZQn10VdhohI1snJUPBRkymjicb6TVGXIiKSVXIyFPLKpwJQt0rXKoiIpMvJUOi6r8JqXasgIpIuJ0NhXPWeAGxbpy0FEZF0ORkKZeUTqKeE2CaFgohIupwMBcxYm6impGlF1JWIiGSV3AwFoKFkChWtNVGXISKSVXI2FJKjZzCejWxtqo+6FBGRrJGzoZA/fhYAa5YtjLgSEZHskbOhUFY9G4AttYsjrkREJHvkbChMnBaEQtu6tyKuREQke2Q0FMzseDNbYmZLzezSHsafZ2Z1ZvZK2J2fyXrSFZWUspZyEluWDdYiRUSyXiJTMzazOHAt8FGgFnjRzO5190XdJv2Tu1+QqTp2ZkNBNaOaV0SxaBGRrJTJLYVDgaXuvszd24DbgVMzuLxd1lw6jQkdq/BUKupSRESyQiZDoRJIvxCgNhzW3SfN7DUzu9PMqnuakZl9xczmm9n8urq6ASvQx8xgJM1s3rBmwOYpIjKUZTIUrIdh3q3/PmCqu88B/gb8tqcZufv17j7X3edWVFQMWIFFE4M2kNYt12mpIiKQ2VCoBdJ/+VcBO9zuzN03untr2Psb4OAM1vMeFVP2AaBplU5LFRGBzIbCi8BMM5tmZvnAWcC96ROY2cS03lOAQf12Hj95Jm0ep6NODeOJiEAGzz5y9w4zuwB4GIgDN7n7QjO7HJjv7vcCF5rZKUAHsAk4L1P19CSeyKM2PonCBp2WKiICGQwFAHd/EHiw27Dvpz2+DLgskzX0ZVNhNWXbVkZZgohI1sjZK5o7tY6axqTkGpIdHVGXIiISuZwPhXj5TAqsnXU1Oq4gIpLzoVBSuTcAG97VaakiIjkfCuOmBqelbl2zJOJKRESil/OhMHZcJY0Uwca3oy5FRCRyOR8KFouxJlFNacM7UZciIhK5nA8FgPrSmUxqW6aG8UQk5ykUgOS4fRhNIxvX1UZdiohIpBQKwIjJcwBY/daCiCsREYmWQgGonDUXgK01r0ZciYhItBQKwOiKiWygjFidWksVkdymUAitKZjOmCadlioiuU2hEGou25OqjpV0tLdFXYqISGQUCqH4xH0ptHZWLVsUdSkiIpFRKIRGTzsAgA3vvBRxJSIi0VEohKpmHkDSjbbVb0RdiohIZBQKocLiEdTGKync9GbUpYiIREahkGZj8QzGbdN9FUQkdykU0rSN3ZtKX0dT45aoSxERiYRCIU1B5X4A1C7RwWYRyU0KhTTjZx4MQP27au5CRHKTQiHNhMmz2OoF+FrdmlNEcpNCIU0sHqcmbyql9bo1p4jkJoVCN/UjZ1KpG+6ISI5SKHTjE/anjCbWrFTjeCKSexQK3YydeTgAaxY9HXElIiKDT6HQzeTZh9LmCdpWzo+6FBGRQZfRUDCz481siZktNbNLdzLdGWbmZjY3k/X0R35BISvypjNy02tRlyIiMugyFgpmFgeuBU4AZgNnm9nsHqYrBS4Ens9ULbtqc9m+TG19m2RHR9SliIgMqkxuKRwKLHX3Ze7eBtwOnNrDdP8OXAW0ZLCWXRKrOpgSa6HmbV3EJiK5JZOhUAnUpPXXhsO6mNmBQLW737+zGZnZV8xsvpnNr6urG/hKuxm315EArH/z2YwvS0Qkm2QyFKyHYd410iwG/Bz4dl8zcvfr3X2uu8+tqKgYwBJ7Vr3HHJq8iNQqHWwWkdySyVCoBarT+quA1Wn9pcC+wBNmtgI4HLg3Gw42xxIJ3i2YyZgtau5CRHJLJkPhRWCmmU0zs3zgLODezpHuXu/u5e4+1d2nAs8Bp7h7Vvw8bxy7H1Pbl9Haui3qUkREBk3GQsHdO4ALgIeBxcA8d19oZpeb2SmZWu5AKZh8CPnWwYqFL0ZdiojIoEn0NYGZzQBq3b3VzI4G5gC/c/c+70Tj7g8CD3Yb9v1epj26PwUPlomzj4TnYcvbz8JBR0VdjojIoOjPlsKfgaSZ7QHcCEwDbs1oVVlgfPVMNjESW/1y1KWIiAya/oRCKtwV9AngF+7+LWBiZsuKnsVi1BTtRUWjDjaLSO7oTyi0m9nZwOeBzusJ8jJXUvbYVr4/U5I1NDVsjroUEZFB0Z9Q+AJwBHCFuy83s2nAHzJbVnYonnYIMXNWvK4WU0UkN/QZCu6+yN0vdPfbzGw0UOruPxmE2iI3df+PAND41t8jrkREZHD0GQpm9oSZjTSzMcCrwM1mdnXmS4veyLHjWB6fyoi1L0RdiojIoOjP7qNR7t4AnA7c7O4HA8dltqzsUTfmYKa3LKS9vS3qUkREMq4/oZAws4nAmWw/0JwzEtM/SIm1suy1Z6IuRUQk4/oTCpcTXJX8jru/aGbTgZy5gfGUA4KNok2Lnoi2EBGRQdDnFc3ufgdwR1r/MuCTmSwqm4ydOJkam0Th6ueiLkVEJOP6c6C5ysz+YmbrzWydmf3ZzKoGo7hssbbsQKZve41kMhl1KSIiGdWf3Uc3E7RuOongJjn3hcNyhk39AKNoZsXirGjAVUQkY/oTChXufrO7d4TdLUDm73STRarC4wp1bzwWcSUiIpnVn1DYYGbnmlk87M4FNma6sGwyYfIs1lJOXq2OK4jI8NafUPgiwemoa4E1wBkETV/kDjNWjTyQKU2v4KlU1NWIiGRMf5q5WOnup7h7hbuPc/fTCC5kyympyUdQzhZq3nkj6lJERDJmd++8dtGAVjEETNz/WADWvKbjCiIyfO1uKNiAVjEEVM6Yw0ZGEVuhxvFEZPja3VDwAa1iCLBYjOWjDmNG4wu6XkFEhq1eQ8HMGs2soYeukeCahZxjM49jDA0sVTtIIjJM9RoK7l7q7iN76Erdvc/mMYajGYedTMqNja88EHUpIiIZsbu7j3JSWcUkluXNoGz1U1GXIiKSEQqFXbRhwlHMaltM/eacun5PRHKEQmEXlc05gYSlWPpczt1aQkRygEJhF+1x4NE0UEzHW49EXYqIyIDrT9PZPZ2FVBM2pz19MIrMJom8fN4ZMZcpm59VkxciMuz0Z0vhauA7BM1mVwEXA78Bbgduylxp2Ss57RgmsIHlb74UdSkiIgOqP6FwvLv/2t0b3b3B3a8HTnT3PwGjd/ZEMzvezJaY2VIzu7SH8V81s9fN7BUz+4eZzd7N1zGophx2MgBrF+jUVBEZXvoTCikzO9PMYmF3Ztq4Xq9sNrM4cC1wAjAbOLuHL/1b3X0/dz8AuIpgqyTrVVTtwbuxakbUPhF1KSIiA6o/oXAO8Flgfdh9FjjXzIqAC3byvEOBpe6+zN3bCHY3nZo+gbs3pPWWMISaz1g7/ij2anmN+s2boi5FRGTA9Kfp7GXufrK7l4fdye6+1N23ufs/dvLUSqAmrb82HLYDM/tnM3uHYEvhwp5mZGZfMbP5Zja/rq6ur5IHxeiDTyffOljy1LyoSxERGTD9OfuoKjzTaL2ZrTOzP5tZVT/m3VNLqu/ZEnD3a919BvBd4Hs9zcjdr3f3ue4+t6IiO+4EOvOgj1DHaOJv6XoFERk++rP76GbgXoJG8CqB+8JhfakFqtP6q4DVO5n+duC0fsw3K1gszvLyj7B30ws0NzX0/QQRkSGgP6FQ4e43u3tH2N0C9Ofn+ovATDObZmb5wFkE4dLFzGam9X4ceLufdWeF0gNPp9haWfz3u6IuRURkQPQnFDaY2blmFg+7c4E+G/5x9w6CA9EPA4uBee6+0MwuN7NTwskuMLOFZvYKwd3cPr+bryMSsw47ni2Uklp0X9SliIgMiP40gf1F4Brg5wTHBJ4BvtCfmbv7g8CD3YZ9P+3xN/pdaRaKJ/J4e/RR7L3pMVq2baWwqDjqkkRE3pf+nH200t1PcfcKdx/n7qcBpw9CbUNC4ZxPUGrbWPyMthZEZOjb3QbxLhrQKoawvY48iUaKaH39nqhLERF533Y3FHo63TQn5RUU8dbID7Dnlqdoa2uLuhwRkfdld0NhyFx5PBgSqVZG08iip7ULSUSGtl5DoZcmsxvMrJHgmgUJ7X3BPBoooe2lW6MuRUTkfen17CN3Lx3MQoay/MJiXiv/KPvVPUj95k2MGj0m6pJERHaL7rw2QEYf8XmKrI1Fj/0+6lJERHabQmGATD/waGpjkyh9846oSxER2W0KhQFisRirp5zGvu2vs/KdRVGXIyKyWxQKA2jasV8EoOaJW6ItRERkNykUBlBF1UwWFRzA5Np7SSVTUZcjIrLLFAoDrGWfM6n2NSx84X+jLkVEZJcpFAbY3secy1YKaH62P7ecEBHJLgqFAVY0YhSLyk/kwPpHqVu3s3sKiYhkH4VCBkz46L9QYO0seeh/oi5FRGSXKBQyoGrPg1lcsD/TV9xOe3t71OWIiPSbQiFDOuaezyTqeOXRP0VdiohIvykUMmT2R85mvY0l/6Uboy5FRKTfFAoZEk/k8e60s9i/7SWWLnop6nJERPpFoZBBs074Z9o8wfpH/zvqUkRE+kWhkEGjKip5Y8xx7L/hAZ2eKiJDgkIhwyaceCkl1sqb91wVdSkiIn1SKGTYpJkH8mrpUey/6k9s3rQh6nJERHZKoTAIRn/sUkbaVl6/+z+jLkVEZKcUCoNg8r4fYGHxoez77u+pr98SdTkiIr1SKAySouO+yxhr5LV7fhl1KSIivVIoDJLpBx3HmwVz2HPZzTQ1N0VdjohIjzIaCmZ2vJktMbOlZnZpD+MvMrNFZvaamT1qZlMyWU/U8o65lHFsZsEdOhNJRLJTxkLBzOLAtcAJwGzgbDOb3W2yl4G57j4HuBMY1t+WMw77OAuLD+WA5TdQt25N1OWIiLxHJrcUDgWWuvsyd28DbgdOTZ/A3R93961h73NAVQbryQqjT72SUray+I5/i7oUEZH3yGQoVAI1af214bDefAl4KIP1ZIVJex7Cq+UncFjdnSx/e1HU5YiI7CCToWA9DPMeJzQ7F5gL/KyX8V8xs/lmNr+urm4AS4zGtE/9GDdj7d3/L+pSRER2kMlQqAWq0/qrgPc0AGRmxwH/FzjF3Vt7mpG7X+/uc919bkVFRUaKHUxlE6ayaPK5HNH8N1594YmoyxER6ZLJUHgRmGlm08wsHzgLuDd9AjM7EPg1QSCsz2AtWWf2p37AZkaS//B3aGvviLocEREgg6Hg7h3ABcDDwGJgnrsvNLPLzeyUcLKfASOAO8zsFTO7t5fZDTuFpaNZc/j32Tv5Fs/MU/MXIpIdzL3H3fxZa+7cuT5//vyoyxgY7iy56mgmbH2bhvOfobp6atQVicgwZWYL3H1uX9PpiuYomTHmzGspopWa2y5iqAW0iAw/CoWIVUzbl4UzvsiRWx/luUfvirocEclxCoUsMOesy1nt5VT//RI2bNwYdTkiksMUClkgnl9E8hPXM5GNLL7pa9qNJCKRUShkieoDjmXh9PP5UPPDPHn3jVGXIyI5SqGQRfY750qW5c9i/1d+wLJlb0ddjojkIIVCFrFEPqPOuZlCa2PLrV+mpa096pJEJMcoFLLM2Cn7smLu9zio42We+s3FOr4gIoNKoZCF9v74hSys+Dj/VHcLT9z326jLEZEcolDIRmbsff4NrMifydwFl/L6qy9GXZGI5AiFQpaKFRQz5ovzSFoeJX85j3V1G6IuSURygEIhi42cMJ3Gk3/DZF9Nza/PoL6pOeqSRGSYUyhkueqDj2f5EVcyt+NlXrvmHJ2RJCIZpVAYAmYe/zUWz/4GH2p5nCf/5+skUzojSUQyQ6EwROz9qR+yqOrTfGzLPB66/jJSCgYRyQCFwlBhxuwvXseSscdx0trreOBXl2mLQUQGnEJhKInF2fPrt7Ok/KOcvP46Hvqfi+lIpqKuSkSGEYXCUBPPY8+v3c6ScSdw0oYb+Ou136RV93gWkQGiUBiK4gn2/OofeWviyZy06bc8cfXn2dK0NeqqRGQYUCgMVbE4s778O96e+SU+tu1+Fv38FGrW6gI3EXl/FApDWSzGzHOuZsVhP+Sw5Hzqf/UxXl74ZtRVicgQplAYBqae8E3Wn3AjM6ihct7HuO/eO9W6qojsFoXCMDHxsE+S+tLf8PwRnLDgy/z5msto3NYWdVkiMsQoFIaRkuo5jLvoaWrGHc0ZG6/j+R+fwIKFS6IuS0SGEIXCMGNFZUz7+l3UHPo9joq9wpR5xzHvD7+mpT0ZdWkiMgQoFIYjM6pP/A4d5z9BW/F4zlx6CU/+9AxeXKj7PovIzikUhrHiqv2Y9O1nWLnP1zm240n2mHc0t//qR9Q1bIu6NBHJUgqF4S6Rz+RP/ZiO85+kedQenLX2Z6y6+ijuuu8e7VISkffIaCiY2fFmtsTMlprZpT2MP8rMXjKzDjM7I5O15LrCqv2o+tYTrD/maqbF1nP6gs/xjx+fxANPPK2G9USkS8ZCwcziwLXACcBs4Gwzm91tspXAecCtmapD0pgx7qgvMeq7b1A751/4YGoBH338ZO6/8izue/JZ2jrUuJ5IrsvklsKhwFJ3X+bubcDtwKnpE7j7Cnd/DdC30WAqKKXq9B9RcNGrrJ1xJid2/I0THjuRR648nT8//BiNLbq7m0iuymQoVAI1af214bBdZmZfMbP5Zja/rq5uQIoTsJETmfy5X5H41mus2etzHJf6B5989hO8/OPj+N3vb+Sd9Y1RlygigyyToWA9DNutndfufr27z3X3uRUVFe+zLOnORlVSffYvKbh4MWsPuogDEiv53DsXwTWH8Lv/vIi7//ESTa1qnlskFyQyOO9aoDqtvwpYncHlyfs1ooIJp/wATryMhgXzKHn613yu4UbaH7mFJx85kBWTPk7Voafx4X2mUJQfj7paEckAy1TDaWaWAN4CjgVWAS8Cn3H3hT1Mewtwv7vf2dd8586d6/Pnzx/gaqU3XreEdU/eSPGbdzKyYyPNXsATfjC1k/6J8QeeyFH7TmNMSX7UZYpIH8xsgbvP7XO6TLamaWYnAr8A4sBN7n6FmV0OzHf3e83sEOAvwGigBVjr7vvsbJ4KhYikkiSX/Z26526jdPlDlCTrafU8nkntw9tlHySx57Hsv98BzKkqIy+uy19Esk1WhEImKBSyQLKD1LvPsPGlu8lb+lfKWlYB8G5qHM/ZHDaMPYSCGUfG2bGTAAAMCUlEQVSy16zZHDC5jBEFmdxLKSL9oVCQweEOG5ey9c1HaFz4CKPWPUdhKrg1aK2XsyA1i1XFe9M+8SDGzDiEPasq2GtiKSML8yIuXCS3KBQkGskOWPcG2955msa3nqJo3UuUtq0HoN3jLPVKFvpUVhXuQeuY2RRU7ktVZTUzxo1g2tgSRhUrLEQyQaEg2aNhDb5qPs3LX6S15hUKN75BSdvGrtF1PoqlqUqW+wRW51XRUjoNHz2N4vHTmVQ+mkllRUwaVciksiJKtCtKZLcoFCS7Na6D9YtIrl1Ic81rpOreprBhGYXtW7omSbmxltGs8nJqvYJVXs6meAWtxRNJlVYSHzWJ4rIKyksLKB9RwJiS/K5udHE+xflxzHq6XEYk9/Q3FPSzS6JROh5KxxOf8RFGpg/fugk2vgObl+MblzFq/VJGbFrJ7MYVFG57nrh3wDaCbj20eYL1lLHBR1Hno1jkI9nAKDb6SLZYGW0Fo0kWjsGLRkPRaAqLSyktyqO0MI/SwgSlBYmuxyMKEpQUJCjOj1NSkKAoP05xXpyEzqaSHKJQkOxSPCboqg8hDpSkj0sloWkd1K+ChlpoWENe41rGN6xlbP0aZjXXEd9aQ17rJmIeNgueBJrDDmgjQQMlbPES6r2EBi+mgRJWezGNFNPoxTRSRKMX0UwRzRSy1QtpsUIKS0ZCfjGeKMIShRTkJSjIi5Efj3X9zU/EyIsHXUHa47yEdY3PD4flJzqnNxKxGIkd/hrxWNAfj0E8FiNuRjwcFzMjZhCPGbGwP26GdQ4Lx2tLSXaVQkGGjlgcRk4KOg4BgrZU8sKuSyoFLVugeQM018G2TbBtM2zdRP62zZS3bGHsti2ktm4mta0eWlZjLfXE2hqJ+U6a82gPu1CLFdBGAa2WTyv5tHg+LeTR4vk0JfNoIY828mjxPNpI0Eo+zSTYRB5tnkcrCdrCado9TjsJ2kjQQfC4nQQdHg/743SQoD0c1+Gdw7YPTxKngxjerfWaeMxIpRwMChKxrtZwS/ITmMHWtiBARxXlUb+tHTMYXZzP5q1tAJSPKCBmRl1TKwaMG1nA+oZWACrLilhdH9y0qXp0MWZQsynonzK2mJWbgjPRZlSMwCx4v3raYd09una2U9sIwq5zfmbW9fz0DLTuc7Xty4l1Pt+2T9dbfnbOf2f5mj6qtyDePo/ttacvt7c9+ek1nn3YZD48K7NN/SgUZPiJxbZvcVTM6nESI7iicofGOtyhowVaGqC1Edqagq61Cdqboa0Z2rZC+1Zo30ZhxzYK27dBewt09PC3Yyt0tOAdrcF8k63Q0YYlWzP68lPEcIuTtAQpS5Cy+PaOOCmL4cRJdg4rCh47MZKFcVLEguEFsfBxLHje6O39PiZGMpwuOcaCZRIj5UZqjJEkTsrjJEfHSRLDO4wkwXOwWFijBcskhmO4BX9TxMP+7Y+DmmOkCJaVctveYWnjtj8O/sZIWTA85Z2vxXCPkXRIhvNNerDewEiZhXUEy0laDPeg/s75bq8pXN/h9O4EtWPvSZGUO+6wYmOw2Tp5TDEA724MgnPK2OIe38/OsHCchm2Zb8FYoSDSyQzyioKudPzAzbb7AHdItgVB0dEKyfagv6trh1RHt8ftkGoPp0173DWuo6uLhf3xzmFd45PgybRpk2HXkTY82W3atvBvKvybTPub2rEfD4elts8v2R4M2722MIc+i723Kwz/bguDY0TY397DtGbBj5zO/tilQGbvR6ZQEBlsZpAoCLpc4b49QDyVFiiptM63j3/PuLTndIbPDv3eLZh6ef57xiXD5fr2x33No3N4eh1d80htr6f7ND3WmrbcHWqiW33huOIxGX+rFAoiknlmEE+gr5zsp3PtRESki0JBRES6KBRERKSLQkFERLooFEREpItCQUREuigURESki0JBRES6DLn7KZhZHfDubj69HNgwgOUMVVoP22ldBLQeAsN5PUxx9z5b0xtyofB+mNn8/txkYrjTethO6yKg9RDQetDuIxERSaNQEBGRLrkWCtdHXUCW0HrYTusioPUQyPn1kFPHFEREZOdybUtBRER2QqEgIiJdciYUzOx4M1tiZkvN7NKo6xksZlZtZo+b2WIzW2hm3wiHjzGzR8zs7fDv6KhrHQxmFjezl83s/rB/mpk9H66HP5lZftQ1ZpqZlZnZnWb2Zvi5OCIXPw9m9q3wf+INM7vNzApz8fPQXU6EgpnFgWuBE4DZwNlmNjvaqgZNB/Btd98bOBz45/C1Xwo86u4zgUfD/lzwDWBxWv9PgZ+H62Ez8KVIqhpcvwT+6u57AfsTrI+c+jyYWSVwITDX3fcF4sBZ5ObnYQc5EQrAocBSd1/m7m3A7cCpEdc0KNx9jbu/FD5uJPgCqCR4/b8NJ/stcFo0FQ4eM6sCPg7cEPYbcAxwZzjJsF8PZjYSOAq4EcDd29x9Czn4eSC4N2iRmSWAYmANOfZ56EmuhEIlUJPWXxsOyylmNhU4EHgeGO/uayAIDmBcdJUNml8AlwCpsH8ssMXdO8L+XPhcTAfqgJvD3Wg3mFkJOfZ5cPdVwH8AKwnCoB5YQO59Ht4jV0LBehiWU+fimtkI4M/AN929Iep6BpuZnQSsd/cF6YN7mHS4fy4SwEHAde5+INDMMN9V1JPwmMmpwDRgElBCsHu5u+H+eXiPXAmFWqA6rb8KWB1RLYPOzPIIAuGP7n5XOHidmU0Mx08E1kdV3yD5AHCKma0g2H14DMGWQ1m4+wBy43NRC9S6+/Nh/50EIZFrn4fjgOXuXufu7cBdwJHk3ufhPXIlFF4EZoZnFuQTHFC6N+KaBkW43/xGYLG7X5026l7g8+HjzwP3DHZtg8ndL3P3KnefSvD+P+bu5wCPA2eEk+XCelgL1JjZnuGgY4FF5NjngWC30eFmVhz+j3Suh5z6PPQkZ65oNrMTCX4ZxoGb3P2KiEsaFGb2QeDvwOts35f+rwTHFeYBkwn+QT7l7psiKXKQmdnRwMXufpKZTSfYchgDvAyc6+6tUdaXaWZ2AMHB9nxgGfAFgh+IOfV5MLMfAp8mOEPvZeB8gmMIOfV56C5nQkFERPqWK7uPRESkHxQKIiLSRaEgIiJdFAoiItJFoSAiIl0UCiIi0kWhINIPZnZAeK1LZ/8pA9UEu5l908yKB2JeIu+XrlMQ6QczO4+gmeULMjDvFeG8N+zCc+LunhzoWkS0pSDDiplNDW8c85vwBir/a2ZFvUw7w8z+amYLzOzvZrZXOPxT4Y1XXjWzp8KmUS4HPm1mr5jZp83sPDO7Jpz+FjO7LryZ0TIz+7CZ3RTWcUva8q4zs/lhXT8Mh11I0CDb42b2eDjsbDN7Pazhp2nPbzKzy83seeAIM/uJmS0ys9fM7D8ys0Yl57i7OnXDpgOmEjRbcEDYP4+gqYKepn0UmBk+PoygPSQImgSpDB+XhX/PA65Je25XP3ALQdMIRtDyZgOwH8GPrgVptYwJ/8aBJ4A5Yf8KoDx8PImgmYkKghZNHwNOC8c5cGbnvIAlbN/aL4t63asbHp22FGQ4Wu7ur4SPFxAExQ7CpsSPBO4ws1eAXwMTw9FPA7eY2ZcJvsD74z53d4JAWefur7t7CliYtvwzzewlgjZ19iG4C2B3hwBPeNB6ZwfwR4Kb4gAkCVq7hSB4WoAbzOx0YGs/6xTZqUTfk4gMOekNmCWBnnYfxQhuqHJA9xHu/lUzO4zgLm2vhA3I9XeZqW7LTwEJM5sGXAwc4u6bw91KhT3Mp6d7PHRq8fA4grt3mNmhBK17ngVcQNAcuMj7oi0FyUke3GhouZl9CoImxs1s//DxDHd/3t2/D2wguBdHI1D6PhY5kuCGNvVmNp4db+iSPu/ngQ+bWXl4b/GzgSe7zyzc0hnl7g8C3wT6E1wifdKWguSyc4DrzOx7QB7BcYFXgZ+Z2UyCX+2PhsNWApeGu5p+vKsLcvdXzexlgt1Jywh2UXW6HnjIzNa4+0fM7DKCdv0NeNDde2rTvxS4x8wKw+m+tas1ifREp6SKiEgX7T4SEZEu2n0kw56ZXUtwj+Z0v3T3m6OoRySbafeRiIh00e4jERHpolAQEZEuCgUREemiUBARkS7/H6Wbbf90AEPYAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "cvresult = pd.DataFrame.from_csv('1_nestimators.csv')\n",
    "        \n",
    "# plot\n",
    "test_means = cvresult['test-logloss-mean']\n",
    "test_stds = cvresult['test-logloss-std'] \n",
    "        \n",
    "train_means = cvresult['train-logloss-mean']\n",
    "train_stds = cvresult['train-logloss-std'] \n",
    "\n",
    "x_axis = range(0, cvresult.shape[0])\n",
    "        \n",
    "pyplot.errorbar(x_axis, test_means, yerr=test_stds ,label='Test')\n",
    "pyplot.errorbar(x_axis, train_means, yerr=train_stds ,label='Train')\n",
    "pyplot.title(\"XGBoost n_estimators vs Log Loss\")\n",
    "pyplot.xlabel( 'n_estimators' )\n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig( 'n_estimators4_1.png' )\n",
    "\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "93"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cvresult.shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
